fix(eval): make the synthetic fundamentals gate real — post-#201 review - #202
Merged
Merged
Conversation
Review of #201 found the committed gates had never been run: 'asa verify' was red on main (3 failing default-manifest checks) and the synthetic manifest failed 52 of 140 checks. This commit makes both gates green with honest measurements and records the first accuracy baseline. Generator (build_synthetic_corpus.py): - Engineered band-disjoint one-shots replace product drum samples in fixtures: the kick's pitch-sweep onset/click landed in the snare band (16 kicks counted as snares) and product hats never cleared the hat detector's floor. Kick = 50 Hz sine, 20 ms fade-in (sharper attacks splatter into the snare band), 0.20 s length (its low-band tail otherwise turns -64 dB snare residue into countable local maxima); snare/hat = band-passed bursts with an extra HP pass. - Grid clips are a bare accented-kick pulse: 'and' hats bin ambiguously in analyze_time_signature's ±half-beat onset counting and flipped 4/4 reads to 3/4 or 6/8. Swing clips keep swung hats but check BPM only. - Chord labels computed from pitch classes, flat-spelled to match the Viterbi vocab (hardcoded sharp table scored 0.25 on F# major). - Bass fixture gets --transcribe (was missing → guaranteed noteF1 0). - expected blocks carry active checks only; ground-truth-for-later (swing %, hit times) moves to a separate 'truth' key. - Dropped the [:28] truncation that silently discarded multi_F_major; one-shots carry their own seeds (order-independent determinism). Harness (fundamentals_evaluation.py): - Enharmonic pitch-class folding for key + chord comparisons ('C# Minor' == 'Db minor', 'Gb' == 'F#'); relative-key table replaced with pitch-class math. - knownGaps: per-track check names that still run and report scores but don't gate — measured baseline weaknesses stay visible as improvement targets (8 at baseline: odd-meter detection + downstream downbeats, tempo octave at 174, 7/8 tempo wobble). Trust layer (fundamentals_quality.py): - bpmAgreement=True with confidence >= 0.5 is authoritative: a BPM measured exactly right and cross-confirmed by Percival was being hedged as 'cross-check not strong enough'. CI: the default-manifest fundamentals gate (render + --fail-on-skip) now runs in the backend job — #201 merged green because CI never executed the gate it shipped. Baseline: audits/accuracy-baseline-2026-07-03.md — 29 clips, 54/54 active checks green, 8 knownGaps quantified. Backend suite: 1283 tests OK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T4wfz87k6kzJqkKLKZE7YL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review-and-fix of #201 (Codex's PR-A1 implementation of the accuracy program). The structure was right, but neither committed gate had ever been run:
asa verify's backend leg was red on main (3 failing default-manifest checks) and the committed synthetic manifest failed 52 of 140 checks when executed. CI never caught it because the workflow only runsunittest discover.What was broken (all verified by running the gates)
drum_stem_known_counts: snare detector counted the 16 kicks (kick click/pitch-sweep energy lands in the 120–2000 Hz band, and Codex placed snares on kick beats so coincident onsets merge); hat detector found 4–8 of 32 hats (below its floor). Counts read 15 and 8 vs truths 8 and 32.four_on_floor_clear_128: BPM measured exactly right (128.0, Percival 128.4, agreement) yet failedquality:tempo— 4-bar clips cap the extractor's confidence below the 0.7 bar.mono_bass_transcriptionmissing--transcribe(noteF1 = 0), chord fixtures spelled sharps against the Viterbi's flat vocab (F# major scored 0.25), key comparison lacked enharmonic folding (Db minor≠C# Minor), BPM checked on transient-free chord clips (read 61–98 vs 96–140 truths), spec feat(ui): re-analyze CTA for legacy Session Musician runs #29 silently dropped by a[:28]slice, ground-truth-for-later (hitTimes,swingPercent) embedded insideexpected.Fixes
Generator — engineered band-disjoint one-shots (kick = 50 Hz sine with a load-bearing 20 ms fade-in and 0.20 s length; snare/hat = double-filtered noise bursts); grid clips are a bare accented-kick pulse (off-beat hats bin ambiguously in
analyze_time_signature's ±half-beat counting and flipped 4/4 reads to 3/4 or 6/8); swing clips keep swung hats but check BPM only; chord labels computed from pitch classes, flat-spelled; per-kindexpectedblocks with everything else undertruth;--transcribeon the bass fixture; all 29 specs render; order-independent determinism.Harness — enharmonic pitch-class folding for key + chord label comparison;
knownGaps: per-track check names that still run and report scores but don't gate (checksInformationalin the summary), keeping measured baseline weaknesses visible as improvement targets.Trust layer —
bpmAgreement=Truewith confidence ≥ 0.5 is nowauthoritative: a BPM measured exactly right and cross-confirmed by an independent estimator was being hedged as "cross-check is not strong enough", which was factually wrong. Covered by new tests; golden unaffected.CI — the default-manifest fundamentals gate (render +
evaluate_fundamentals.py --fail-on-skip) now runs in the backend job, closing the hole that let #201 merge with a red gate.Results
asa verify/ CI path): 15/15 checks green.audits/accuracy-baseline-2026-07-03.md, the accuracy program's "before" column.🤖 Generated with Claude Code
https://claude.ai/code/session_01T4wfz87k6kzJqkKLKZE7YL
Generated by Claude Code